Jenkins plugin to run dynamic agents in a Kubernetes cluster.

Based on the Scaling Docker with Kubernetes article, automates the scaling of Jenkins agents running in Kubernetes.

The plugin creates a Kubernetes Pod for each agent started, and stops it after each build.

Agents are launched as inbound agents, so it is expected that the container connects automatically to the Jenkins controller. For that some environment variables are automatically injected:

Tested with jenkins/inbound-agent, see the Docker image source code.

It is not required to run the Jenkins controller inside Kubernetes.

Fill in the Kubernetes plugin configuration. In order to do that, you will open the Jenkins UI and navigate to Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds -> Add a new cloud -> Kubernetes and enter the Kubernetes URL and Jenkins URL appropriately, unless Jenkins is running in Kubernetes in which case the defaults work.

Supported credentials include:

If you check WebSocket then agents will connect over HTTP(S) rather than the Jenkins service TCP port. This is unnecessary when the Jenkins controller runs in the same Kubernetes cluster, but can greatly simplify setup when agents are in an external cluster and the Jenkins controller is not directly accessible (for example, it is behind a reverse proxy or a ingress resource). See JEP-222 for more.

To test this connection is successful you can use the Test Connection button to ensure there is adequate communication from Jenkins to the Kubernetes cluster, as seen below

In some exceptional cases, agent pods can be left behind, with no declared Jenkins agent in the controller. They will try to reconnect over and over, until something deletes them.

The plugin provides a garbage collection mechanism to clean up these pods. As it has been introduced recently, and generates extra load on the Kubernetes API server, it is disabled by default.

Feel free to enable it and provide feedback about this functionality.

In addition to that, in the Kubernetes Pod Template section, we need to configure the image that will be used to spin up the agent pod. The Jenkins agent runs by default in a container named jnlp (historical name, kept for backward compatibility).

You may use it, use a different agent container name and/or provide your own container image.

To avoid drift between the Jenkins controller and your agent, we recommend not to include the agent JAR in your image, but instead use the Inject Jenkins agent in agent container checkbox.

The provided container image must have a JRE installed that is compatible with the Java version required by the Jenkins controller.

In the ‘Kubernetes Pod Template’ section you need to specify the following (the rest of the configuration is up to you): Kubernetes Pod Template Name - can be any and will be shown as a prefix for unique generated agent’ names, which will be run automatically during builds Docker image - the docker image name that will be used as a reference to spin up a new Jenkins agent, as seen below

Note: If your Jenkins controller is outside the cluster and uses a self-signed HTTPS certificate, you will need some additional configuration.

Clouds can be configured to only allow certain jobs to use them.

To enable this, in your cloud's advanced configuration check the Restrict pipeline support to authorized folders box. For a job to then use this cloud configuration you will need to add it in the jobs folder's configuration.

The Kubernetes plugin allocates Jenkins agents in Kubernetes pods.

Within these pods, there is always one special container that is running the Jenkins agent.

Its name defaults to jnlp, but you can override the name with one of your choosing in pod template configuration.

Other containers can run arbitrary processes of your choosing, and it is possible to run commands dynamically in any container in the agent pod using the container step (pipeline only).

Pod templates defined using the user interface declare a label. When a freestyle job or a pipeline job using node('some-label') uses a label declared by a pod template, the Kubernetes Cloud allocates a new pod to run the Jenkins agent.

It should be noted that the main reason to use the global pod template definition is to migrate a huge corpus of existing projects (including freestyle) to run on Kubernetes without changing job definitions. New users setting up new Kubernetes builds should use the podTemplate step as shown in the example snippets here.

The podTemplate step defines an ephemeral pod template. It is created while the pipeline execution is within the podTemplate block. It is immediately deleted afterwards. Such pod templates are not intended to be shared with other builds or projects in the Jenkins instance.

The following idiom creates a pod template with a generated unique label (available as POD_LABEL) and runs commands inside it.

Commands will be executed by default in the designated agent container, where the Jenkins agent is running.

This will run in the agent container:

Find more examples in the examples dir.

The default agent image used can be customized by adding it to the template

or with the yaml syntax. Pretty much any field from the pod model can be specified through the yaml syntax.

Multiple containers can be defined for the agent pod, with shared resources, like mounts. Ports in each container can be accessed as in any Kubernetes pod, by using localhost.

One container must run the Jenkins agent. If unspecified, a container named jnlp will be created with the inbound-agent image. The Jenkins agent requires a JRE to run, so you can avoid the extra container by providing a name using the agentContainer. To get the Jenkins agent injected, you will also need to set agentInjection to true, and leave the command and argument fields empty for this container. The container specified by agentContainer will be the one where shell steps (or any other step running remote commands on the agent) will run on.

To execute commands in another container part of the pod (different from the one running the Jenkins agent), you can use the container step.

or

The variable POD_CONTAINER contains the name of the container in the current context. It is defined only within a container block.

You can use the retry step to automatically try the whole build stage again with a fresh pod in case of fatal infrastructure problems. (For example: cluster backup & restore; node pool used for agents drained and upgraded.)

will rerun the whole node block (using the same pod definition) in case the first attempt fails for a qualifying reason traceable to loss of the pod (not routine problems such as compilation errors or OutOfMemoryError).

For Declarative Pipeline, just add the retries option, as shown below.

Pod templates are used to create agents. They can be either configured via the user interface, or in a pipeline, using the podTemplate step. Either way it provides access to the following fields:

Container templates are part of pod. They can be configured via the user interface or in a pipeline and allow you to set the following fields:

By default, the agent connection timeout is set to 1000 seconds. It can be customized using a system property. Please refer to the section below.

In order to support any possible value in Kubernetes Pod object, we can pass a yaml snippet that will be used as a base for the template. If any other properties are set outside the YAML, they will take precedence.

You can use readFile or readTrusted steps to load the yaml from a file. Also note that in declarative pipelines the yamlFile can be used (see this example).

pod.yaml

Jenkinsfile

See Defining a liveness command for more details.

A pod template may or may not inherit from an existing template.

Depending on fields, the inheritance behaviour can vary. For simple values (strings), the child template will override the parent template. However for complex values (lists, maps), the child template will merge with the parent template.

In the example below, we will inherit from a pod template we created previously, and will just override the version of maven so that it uses Java 21 instead:

Or in declarative pipeline

Note that we only need to specify the things that are different. So, command and arguments are not specified, as they are inherited. Also, the golang container will be added as defined in the 'parent' template.

Field inheritFrom may refer a single pod template or multiple separated by space. In the later case each template will be processed in the order they appear in the list (later items overriding earlier ones). In any case if the referenced template is not found it will be ignored.

Field inheritFrom provides an easy way to compose pod templates that have been pre-configured. In many cases it would be useful to define and compose pod templates directly in the pipeline using groovy. This is made possible via nesting. You can nest multiple pod templates together in order to compose a single one.

The example below composes two different pod templates in order to create one with maven and docker capabilities.

This feature is extra useful, pipeline library developers as it allows you to wrap pod templates into functions and let users nest those functions according to their needs.

For example one could create functions for their podTemplates and import them for use. Say here's our file src/com/foo/utils/PodTemplates.groovy:

Then consumers of the library could just express the need for a maven pod with docker capabilities by combining the two, however once again, you will need to express the specific container you wish to execute commands in. You can NOT omit the node statement.

Note that POD_LABEL will be the innermost generated label to get a node which has all the outer pods available on the node, as shown in this example:

In scripted pipelines, there are cases where this implicit inheritance via nested declaration is not wanted or another explicit inheritance is preferred. In this case, use inheritFrom '' to remove any inheritance, or inheritFrom 'otherParent' to override it.

Declarative agents can be defined from yaml

or using yamlFile to keep the pod template in a separate KubernetesPod.yaml file

Note that it was previously possible to define containerTemplate but that has been deprecated in favor of the yaml format.

Run steps within a container by default. Steps will be nested within an implicit container(name) {...} block instead of being executed in the agent container.

Run the Pipeline or individual stage within a custom workspace - not required unless explicitly stated.

Unlike scripted k8s template, declarative templates do not inherit from parent template. Since the agents declared at stage level can override a global agent, implicit inheritance was leading to confusion.

You need to explicitly declare the inheritance if necessary using the field inheritFrom.

In the following example, nested-pod will only contain the maven container.

If you use the containerTemplate to run some service in the background (e.g. a database for your integration tests), you might want to access its log from the pipeline. This can be done with the containerLog step, which prints the log of the requested container to the build log.

Also see the online help and examples/containerLog.groovy.

Please read Features controlled by system properties page to know how to set up system properties within Jenkins.

OpenShift runs containers using a random UID that is overriding what is specified in Docker images. For this reason, you may end up with the following warning in your build

At the moment the jenkinsci agent image is not built for OpenShift and will issue this warning.

This issue can be circumvented in various ways:

See this example configuration.

OpenShift 3 is based on an older version of Kubernetes, which is not anymore directly supported since Kubernetes plugin version 1.26.0.

To get agents working for Openshift 3, add this Node Selector to your Pod Templates:

You can run pods on Windows if your cluster has Windows nodes. See the example.

Create and start minikube

The client certificate needs to be converted to PKCS, will need a password

Validate that the certificates work

Add a Jenkins credential of type certificate, upload it from ~/.minikube/minikube.pfx, password secret

Fill Kubernetes server certificate key with the contents of ~/.minikube/ca.crt

Create a cluster

and note the admin password and server certificate.

Or use Google Developer Console to create a Container Engine cluster, then run

the last command will output kubernetes cluster configuration including API server URL, admin password and root certificate

First watch if the Jenkins agent pods are started. Make sure you are in the correct cluster and namespace.

If they are in a different state than Running, use describe to get the events

If they are Running, use logs to get the log output

If pods are not started or for any other error, check the logs on the controller side.

For more detail, configure a new Jenkins log recorder for org.csanchez.jenkins.plugins.kubernetes at ALL level.

To inspect the json messages sent back and forth to the Kubernetes API server you can configure a new Jenkins log recorder for okhttp3 at DEBUG level.

To debug this you need to set -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true system property and then restart the pipeline. Most likely in the console log you will see the following:

Usually this happens when UID of the user in agent container differs from the one in another container(s). All containers you use should have the same UID of the user, also this can be achieved by setting securityContext:

Using WebSockets is the easiest and recommended way to establish the connection between agents and a Jenkins controller running outside the cluster. However, if your Jenkins controller has HTTPS configured with self-signed certificate, you'll need to make sure the agent container trusts the CA. To do that, you can extend the jenkins/inbound-agent image and add your certificate as follows:

Then, use it as the agent container for the pod template as usual. No command or args need to be specified.

Notes:

Support for using WebSockets with JDK 11 was added in the Remoting v4.11, so make sure your base image is new enough. See here for more information.

When using the WebSocket mode, the -disableHttpsCertValidation on the jenkins/inbound-agent becomes unavailable, as well as -cert, and that's why you have to extend the docker image.

You need to change from something like:

To something like:

Integration tests will use the currently configured context auto-detected from kube config file or service account.

Run mvn clean install and copy target/kubernetes.hpi to Jenkins plugins folder.

Set up your $KUBECONFIG however you like, for example

then

Alternately, you can run everything like in CI:

You can also run interactively after setting up the tunnel:

Docker image for Jenkins, with plugin installed. Based on the official image.

The example configuration will create a stateful set running Jenkins with persistent volume and using a service account to authenticate to Kubernetes API.

A local testing cluster with one node can be created with minikube

You may need to set the correct permissions for host mounted volumes

Then create the Jenkins namespace, controller and Service with

Get the url to connect to with

Assuming you created a Kubernetes cluster named jenkins this is how to run both Jenkins and agents there.

Creating all the elements and setting the default namespace

Connect to the ip of the network load balancer created by Kubernetes, port 80. Get the ip (in this case 104.197.19.100) with kubectl describe services/jenkins (it may take a bit to populate)

Until Kubernetes 1.4 removes the SNATing of source ips, seems that CSRF (enabled by default in Jenkins 2) needs to be configured to avoid WARNING: No valid crumb was included in request errors. This can be done checking Enable proxy compatibility under Manage Jenkins -> Configure Global Security

Configure Jenkins, adding the Kubernetes cloud under configuration, setting Kubernetes URL to the container engine cluster endpoint or simply https://kubernetes.default.svc.cluster.local. Under credentials, click Add and select Kubernetes Service Account, or alternatively use the Kubernetes API username and password. Select 'Certificate' as credentials type if the kubernetes cluster is configured to use client certificates for authentication.

Using Kubernetes Service Account will cause the plugin to use the default token mounted inside the Jenkins pod. See Configure Service Accounts for Pods for more information.

You may want to set Jenkins URL to the internal service IP, http://10.175.244.232 in this case, to connect through the internal network.

Set Container Cap to a reasonable number for tests, i.e. 3.

Add an image with

Now it is ready to be used.

Tearing it down

Modify file ./src/main/kubernetes/jenkins.yml with desired limits

Note: the JVM will use the memory requests as the heap limit (-Xmx)